The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also be inventions.
The popularity of touch sensitive surfaces such as touch screen devices, of the type found in many types of tablets and smart phones, has rapidly increased. Touch screens are also being implemented with larger systems, such as laptop and desktop computers, interactive television systems, and the like. As the use of touch screens increase in diverse number of applications, techniques for reliably using touch screens have also grown.
Contemporary touch screens report X/Y coordinates of touch screen contacts to operating systems, interfaces, and various applications. Most capacitive touch screens provide a touch ellipsoid at each point contacted by a finger. The touch ellipsoid has major and minor axes. A vector of the major axis is orientation (which is related to a finger's yaw relative to the touch screen). However, the vector of the major axis is not directional, thus offering two possible yaw indications for a touch blob.
There has been significant research into enhancing interaction on touch screen devices. One approach is to use conventional x-y touch data in combination with spatial or temporal sequences, for example, tap-and-hold and multi-finger or so-called “multi-touch” chording gestures. Examples of this are described in U.S. Patent Publication No. 2007/0177803 entitled “Multi-touch gesture dictionary”, in a paper entitled Shadow Guides: Visualizations for in-situ learning of multi-touch and whole-hand gestures, by Benko et al. and published in the Proceedings ITS '09, in a paper entitled “The design and evaluation of multitouch marketing menus” by Lepinski et al, and published in Proceedings CHI '10, and in a paper entitled “Experimental analysis of mode switching techniques in pen based user inter-faces by Li, et al. in Proceedings CHI '05.
Efforts to determine additional dimensions of information during a touch interaction for example pressure such as are described in a paper entitled “Pressure Marks” published by Ramos et al. in Proceedings CHI 2007, and in a paper entitled “Pressure Widgets” published by Ramos et al. in Proceedings, CHI 2007 and are being implemented in a “force touch” system in the iPhone 6s and iPhone 6S Plus smartphones sold by Apple Computer Company, Cupertino, Calif., USA, shear forces as described for example in a paper entitled “Force Gestures: augmented touch screen gestures using normal and tangential force” published by Heo et al. in Proceedings UIST 2011 and in a paper entitled “One-Point Touch Input of Vector Information from Computer Displays” published by Herot et al. in Proceedings SIGGRAPH 1978, shape of the hands as described for example in a paper entitled “Shape Touch: Leveraging contact on interactive surfaces”, published by Cao, et al. in Proceedings ITS 2008, and in a paper entitled “Touch Tools: Leveraging Familiarity and Skill with Physical Tools to Augment Touch Interaction”, by Harrison et al. in Proceedings CHI 2014, rolling motions of generally stationary fingers as described in a paper entitled “MicroRolls: expanding touch-screen input vocabulary by distinguishing rolls vs. slides of the thumb”, published by Rodaut et al. in Proceedings CHI 2009 and what part of the finger was used to touch the screen as described for example in a paper entitled “TapSense: Enhancing Finger Interaction on Touch Surfaces”, published by Harrison et al. in Proceedings UIST 2011.
It will be appreciated however that enabling more accurate determination of finger yaw during touch events enables more information to be communicated with each individual touch interaction. This, in turn, can have a number of beneficial effects including but not limited to reducing the number of touches, the need for chording and/or spatial sequences and reducing the time required to make an input. The ability to interact with a touch screen using more accurate yaw can for example reduce the need for using spatial sequences and therefore be particularly valuable in applications that have a relatively small touch screen or where only a portion of a larger screen is available for input. Further, the ability to interact with a touch screen using yaw can for example reduce the need for using temporal sequences thereby reducing the amount of time required for interactions with a touch screen. Additionally, in some instances, the ability to interact with a touch screen using more accurate yaw determinations may enable more interactions that more closely emulate familiar physical input modalities that involve twisting or rotating motions.
A variety of approaches for estimating finger yaw have been proposed. For example, some approaches attempt to determine yaw based upon video signals from cameras operating behind or above a display. Examples of this include the system described in a paper entitled “Visual touchpad: a two-handed gestural input device” published by Malik et al., In Proc. ICMI '04, 289-296 and a paper entitled “Empirical evaluation for finger input properties in multi-touch interaction”, published by Wang et al. In Proc. CHI '09. 1063-1072. Systems including using finger-mounted sensors have also been described in a paper entitled “Measurement of finger posture and three-axis fingertip touch force using fingernail sensors” published by Mascaro et al. in IEEE Trans. on Robotics and Automation, 2004.
Closely related to the determination of yaw is the determination of finger pitch. This too can provide an avenue for additional input to be received from a single touch relieving the need for spatial and temporal chording when interacting with a touch sensitive device. In one example, a paper entitled “PointPose: finger pose estimation for touch input on mobile devices using a depth sensor” published by Kratz et al. in proceedings ITS 2013 used a depth camera mounted obliquely to the touch screen to capture finger “rotation and tilt”. Similarly, in a paper entitled “KinectTouch: accuracy test for a very low-cost 2.5D multitouch tracking system” published by Dippon et al. in Proceedings ITS 2011 a depth camera is described but it is mounted above the display. In a paper entitled “Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces, published by Takeoka et al, in Proceedings ITS 2010 uses a series of multiplexed infrared line lasers to create a shallow-field depth sensing touch screen, capable of recovering finger angle. Further, in a paper entitled “AnglePose: robust, precise capacitive touch tracking via 3d orientation estimation, published by Rogers et al. in Proceedings CHI described the use of a 4×6 grid of capacitive-sensing electrodes and a particle filter approach to estimate 3D finger orientation. This setup is used to evaluate how pitch/yaw information can assist in targeting, but the pitch/yaw estimates themselves were never evaluated. Finally, in a paper entitled “The generalized perceived input point model and how to double touch accuracy by extracting finger prints” published by Holtz et al in Proceedings CHI 2010 a commercial-grade fingerprint scanner is described as being used to estimate pitch and yaw based on the fingerprint patch that was visible, which was also used to improve targeting accuracy.
It will be appreciated that all of the above systems rely on special hardware beyond the touch screen to determine pitch and yaw. What are needed therefore are methods and devices that enable a data sensed by a touch screen system to be used to make more accurate determinations of pitch and yaw of a finger or other elongated object in contact therewith.
It will also be understood that there can be a significant amount of noise in x-y reporting touch screen data. This noise can be caused by variations between different types of touch screens, variations within different units of the same type, and variations caused by environmental conditions that may impact the sensitivity of the touch screens.
Accordingly, what are also needed are methods and devices that enable data sensed by an x-y reporting touch screen system to be used to make more accurate determinations of pitch and yaw despite such noise.